Hierarchical Skill Learning for High-Level Planning

Many of the existing techniques for controlling goaldirected agent behavior use one of two primary approaches: heuristic-search planning (HSP) or reinforcement learning (RL). Each has its advantages and disadvantages. For instance, heuristic-search planning does not traditionally learn from previous experience, and can only be applied in domains for which a complete domain model exists. On the other hand, reinforcement learning often performs poorly in new situations until it has gained enough experience to learn an effective policy, and it is difficult to scale RL up to large, complex domains. Both RL and HSP tend to work poorly in domains that require long action sequences. Heuristic-search state-space planning is intractable in such domains, because of the very large search spaces, and reinforcement learning may require exponentially many execution traces to converge. For agents with only low-level primitive actions, such as moving limbs, this makes it intractable to solve problems in complex domains. In order to address both the unique and the shared problems of HSP and RL, I propose a new research direction called skill bootstrapping (SB). The goal of SB is to provide an integrated learning and planning architecture that can improve its performance over time in complex domains. An SB agent starts with a basic set of primitive actions (and their preconditions and effects) as its model of the world. Over the course of solving numerous problems by applying HSP to the primitive actions, SB identifies recurring subgoals, for which it uses RL to create skills that can be applied within the HSP process to solve these subgoals more efficiently. Subgoals can be set by a human supervisor, by a request from the environment, or through a process of exploration. The skills behave as partial policies that can be used reactively, without lengthy deliberative reasoning. Once a new skill is learned, it becomes available for use by the planner along with the other primitive actions, allowing for more compact plans. Additionally, just as future plans can use learned skills, future skills may be built upon lower-level skills. Over the course of the agent’s experience,